Skip to main content

Reply to this post | Go Back
View Post [edit]

Poster: brewster Date: Apr 24, 2003 10:27am
Forum: etree Subject: shows better served as tar files? (and one-at-a-time also supported)

I was wondering what you guys thought about distributing the shows as tar files by default rather than one show at a time. I would think this would be more convenient for most circumstances. Since files can restart if they are broken in the middle, this seems like a good fit even on flakey connections.

unpacking .tar seems to be supported on most platforms (stuffit I think, winzip, os-x and linux of course)

Also, the download manager in browsers overload our servers because it starts multiple downloads, so we limited the number of connections per ip address and this breaks the browser's download manager. therefore people have to use ftp programs most of the time.

Also, it would help with a new download system we are testing called freecache.

there is an untested perl script for unpacking a tar file on the server and just giving a single file for the cases people want to do this
http://www.dc.fit.qut.edu.au/cgi-src/tar.cgi

Therefore, the idea is use TAR'ed shows by default, allow one-at-a-time downloading via a script.

this would allow people using normal browsers to get to this collection much easier than we currently support.

What do you think?

-brewster

Reply to this post
Reply [edit]

Poster: bleblanc Date: Apr 25, 2003 2:06am
Forum: etree Subject: Resume and Sharing?

Two more points/questions.

1. Even though resume is supported, sometimes it breaks if something goes wrong. For files that are 1GB+, this issue will increase in number of occurences and increase in impact to the users. How much more will we see it? We won't know until we test or implement it.

2. If we're now downloading the show in a single try, will this affect Jon's bandwidth sharing tool? In other words, will the software not be able to share out data to other users until the whole download is complete or will it be able to send out data after the first 100K are downloaded?

-Brad

Reply to this post
Reply [edit]

Poster: Jonathan Aizen Date: Apr 25, 2003 3:40am
Forum: etree Subject: Re: Resume and Sharing?

Not sure about resuming and associated problems.

The client we're writing doesn't do typical sharing - it only shares something you're currently downloading. More on all of this soon,

Jon

Reply to this post
Reply [edit]

Poster: Ron G Date: Apr 25, 2003 11:11am
Forum: etree Subject: Re: shows better served as tar files? (and one-at-a-time also supported)

What about getting wget or cURL to work with the archive? I tried to recursively get all the files in a directory using this syntax:

wget -nc -r http://etree07.archive.org/0/audio/sim-uniit2002-09-14.shnf/

but it had problems. Both wget and cURL are availalbe on many many platforms. Perhaps someone should look into those?


This post was modified by Ron G on 2003-04-25 18:11:42

Reply to this post
Reply [edit]

Poster: Jonathan Aizen Date: Apr 24, 2003 11:01am
Forum: etree Subject: Re: shows better served as tar files? (and one-at-a-time also supported)

i think it's a cool idea, but if we're going to still allow one-at-a-time, then we've got double disk consumption on the servers.

the download client i'm working on will allow directories to be downloaded as a "package", allow resumption. i'm aiming for beta first week in may, and release 1.0 in second week of may.

there's another problem with tar files: the user will need twice the amount of disk space for a concert. since these shows are already compressed, here's the effect: let's say you want to download a 1.1gb concert because the tar needs to be uncompressed. the tar file will be just about 1.1gb, then the decompressed files would also be 1.1gb, requiring a total of 2.2gb. for users like me who have just little disk space this wouldn't work.

just my two cents. what do others think?

This post was modified by Jonathan Aizen on 2003-04-24 18:01:19

Reply to this post
Reply [edit]

Poster: Erich Date: Apr 24, 2003 12:37pm
Forum: etree Subject: Re: shows better served as tar files? (and one-at-a-time also supported)

im all for downloading shows in one shot, since it would make things a lot easier to think about. Since you guys are considering making a client for the LMA, then maybe you can also create your own file type. Say tr2002-10-31.lma was a compressed archive of the Tim Reynolds show of that date, and your client would find a logical way of unpacking it to prevent the senario Jon mentioned of taking double space after compression.

Reply to this post
Reply [edit]

Poster: brewster Date: Apr 24, 2003 4:52pm
Forum: etree Subject: Re: shows better served as tar files? (and one-at-a-time also supported)

We would not need to store tar'ed and untar'ed version: just the tar'ed version and there is a cgi script that allows a tar file to be pulled apart at runtime rather than storing it untar'ed.

I am all for a download client. if we can use the browser that might be a good standard alternative.

also, with mirroring and freecache (and I believe bittorrent) it is helpful to have one big file. I have not done the analysis, but I would imagine most downloaders download a full show rather than a single track (especially because they are named with the show sequence rather than the title).

my 2 cents.

-brewster

Reply to this post
Reply [edit]

Poster: Jonathan Aizen Date: Apr 24, 2003 10:08pm
Forum: etree Subject: Re: shows better served as tar files? (and one-at-a-time also supported)

since we would not need to duplicate files on the server end, i think this could be great.

one question: since a cgi is used to pull apart the tar file, what about ftp downloaders?

Reply to this post
Reply [edit]

Poster: brewster Date: Apr 25, 2003 12:53am
Forum: etree Subject: Re: shows better served as tar files? (and one-at-a-time also supported)


I would think ftp downloaders would just download the whole tar file. ftp downloaders would not see the individual songs. that is a downside. I dont know how big a downside.

-brewster

Reply to this post
Reply [edit]

Poster: bleblanc Date: Apr 24, 2003 10:47pm
Forum: etree Subject: Perl Script for One at a Time?

there is an untested perl script for unpacking a tar file on the server and just giving a single file for the cases people want to do this

So, if someone wants to download one track to sample the sound quality, they would have to install Perl and use a script? Let me know if I misunderstood. Figuring out how to use Perl to download one track might be a bit cumbersome for the average user...

I'm not against doing this, I think it's great that everyone gets to figure out how to use a Perl script, but I'm sure we'll get flooded with questions and problems once we do this unless it's very user friendly to do so. - We still get emails once in awhile asking what to do with a "flac" file...

I'm guessing this won't change the size of the show at all, otherwise etree servers would have been doing this a long time ago. Is that correct?

-Brad

This post was modified by bleblanc57 on 2003-04-25 05:47:51

Reply to this post
Reply [edit]

Poster: Jonathan Aizen Date: Apr 24, 2003 11:23pm
Forum: etree Subject: Re: Perl Script for One at a Time?

So, if someone wants to download one track to sample the sound quality, they would have to install Perl and use a script?

If I understood Brewster correctly, I think the Perl script runs as a CGI on the server, which means that, to the downloading user (at least via HTTP), the show appears the same as always. It, on the fly, decompresses the correct file from within the tar file and serves it up to the user. The advantage is that, if the user wanted, he or she could also get it as one package. It is an alternative to using a download manager to queue up the files, and since it is one file, would be easy to resume.

I'm guessing this won't change the size of the show at all, otherwise etree servers would have been doing this a long time ago. Is that correct?

Right. Tar doesn't actually provide any compression from what I've seen (hence people gzip tar balls). It just packages files together. Since the SHNs/FLACs are already compressed (in a format optimized for audio files), even gzip couldn't compress them more, or at least I don't think it could.

This post was modified by Jonathan Aizen on 2003-04-25 06:22:28

This post was modified by Jonathan Aizen on 2003-04-25 06:23:02